Peripheral devices are needed to control operation of various memory elements in three-dimensional memory devices. As the number of stacks in three-dimensional memory devices increases, the height of contact via structures for providing electrical contact to peripheral devices increases. Such an increase in the height of contact via structures makes it difficult to form reliable contact via structures for the peripheral devices.